The present application relates to an image processing apparatus and method by which a global motion representative of deformation of an entire image between two screen images is calculated.
A block matching technique which determines a motion vector between two screen images from image information itself is a technique of an old history.
The block matching technique is a method of calculating a motion vector between two screen images including a reference screen image which is a noticed screen image and an original screen image (hereinafter referred to as target screen image) from which a motion of the reference screen image originates. According to the block matching technique, the motion vector is calculated by calculating a correlation between the reference screen image and the target screen image with regard to blocks of a rectangular region of a predetermined size (a target block and a reference block).
The block matching technique includes two cases including a case wherein the target screen image precedes in time to the reference screen image and another case wherein the reference screen image precedes in time to the target screen image. An example of the former case is motion detection by the MPEG (Moving Picture Experts Group) system, and an example of the latter case is noise reduction by superposition of image frames hereinafter described.
It is to be noted that, in the present specification, the term screen image signifies an image formed from image data of one frame or one field. However, for the convenience of description, in the following description of the present specification, it is assumed that one screen image is formed from one frame. Therefore, one screen image is hereinafter referred to as frame. Accordingly, a reference screen image is hereinafter referred to as reference frame and a target screen image is hereinafter referred to as target frame.
A motion vector detection method by the block matching has been developed particularly in regard to pan-tilt detection and image pickup object tracking of a television camera, moving picture coding of the MPEG system and so forth. In the nineties, application over a wide range including sensorless camera shake correction by superposition of images, noise reduction upon image pickup in low illuminance and so forth has been promoted.
Further, the motion vector detection method by the block matching is applied not only for the image recognition applications and the camera shake correction applications but also for new applications like automatic adjustment of the shutter speed of an image pickup apparatus and double-speed frame rate conversion of a liquid crystal television set.
Incidentally, a motion vector, that is, a local motion vector, of each of a large number of target blocks set in a screen image is detected, and a global motion representative of deformation applied to the entire image between the two screen images is calculated using a large number of local motion vectors detected in this manner. The global motion normally indicates a motion and an amount of the motion of the background of the image.
As a related art, a distortion correction method is disclosed in Japanese Patent Laid-Open No. 2007-221631 (hereinafter referred to as Patent Document 1) wherein one screen image is divided into a large number of blocks and a vector is determined for each of the blocks. Then, the motion vectors determined in this manner are directly used to calculate a global motion. The technique of Patent Document 1 had been used as a camera shake correction technique principally for moving pictures till several years ago within which the number of pixels was small.
With the technique disclosed in Patent Document 1, it is possible to detect a global motion with a hardware scale of a low cost and implement good sensorless or gyroless camera shake correction and noise reduction for moving pictures and still pictures of high picture quality.
However, where one screen image is divided into comparatively large blocks like, for example, approximately 16 blocks, there is a problem that, when a moving image pickup object is included in the image, it is difficult to follow up the moving image pickup object.
Thus, if the number of blocks is increased, for example, to approximately 64, then a large number of local motion vectors regarding a comparatively small image region can be used, and therefore, it becomes possible to follow up the movement of a moving image pickup object to some degree. However, where such a configuration as just described is adopted, a problem occurs with the technique of Patent Document 1 that the hardware scale increases and the cost merit which is an advantage of the technique of Patent Document 1 cannot be made the most of.
Meanwhile, as a method of determining a global motion, a method of applying affine transformation to a plurality of detected local motion vectors has been proposed.
FIG. 34 illustrates a general formula of the affine transformation as (expression 1). Referring to FIG. 34, in the (expression 1), v represents a horizontal component of a motion vector of a target block, w represents a vertical component of the motion vector of the target block, and a, b, c, d, e and f represent affine parameters. In ordinary affine transformation, the affine parameters a, b, c, d, e and f are fixed values, Further, x and y represent a horizontal component and a vertical component of the coordinate of the center of the target block, respectively.
From the affine parameters determined by a convergence mathematical operation process of a global motion and the coordinate of the center of each target block, a motion vector corresponding to the global motion is obtained. Such a motion vector as just mentioned is hereinafter referred to as ideal motion vector. The sum total ε of errors between the ideal vectors and observed motion vectors detected by block matching is represented like an (expression 2) of FIG. 35.
The proposition that a global motion is derived is estimation of the affine parameters a to f which minimize the sum total ε of the errors described above and can be solved, for example, by the least squares method. An (expression 3), another (expression 4) and a further (expression 5) of FIGS. 36, 37 and 38, respectively, illustrate a deriving process of the affine parameters a to f and results of the deriving process.
While the parameters of the affine transformation are calculated comparatively easily in this manner, the effect of this is high. Since the affine transformation is ready not only for parallel movement, rotation and expansion or contraction of an image but also for some degree of deformation, most camera shake, that is, fine correction of the camera work, is covered.
Such affine transformation as described above is disclosed, for example, in Japanese Patent Laid-Open No. 2005-321902 (hereinafter referred to as Patent Document 2).